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Abstract. In this paper we investigate the convergence behavior of a primal-dual 
splitting method for solving monotone inclusions involving mixtures of composite, Lip- 
schitzian and parallel sum type operators proposed by Combettes and Pesquet in [7]. 
Firstly, in the particular case of convex minimization problems, we derive convergence 
rates for the sequence of objective function values by making use of conjugate duality 
techniques. Secondly, we propose for the general monotone inclusion problem two new 
schemes which accelerate the sequences of primal and/or dual iterates, provided strong 
monotonicity assumptions for some of the involved operators are fulfilled. Finally, we 
apply the theoretical achievements in the context of different types of image restoration 
problems solved via total variation regularization. 
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1 Introduction and preliminaries 

The last few years have shown a rising interest in solving structured nondifferentiable 
convex optimization problems within the framework of the theory of conjugate functions. 
Applications in fields like signal and image processing, location theory and supervised 
machine learning motivate these efforts. 

In this article we investigate and improve the convergence behavior of the primal- 
dual monotone + skew splitting method for solving monotone inclusions which was 
proposed by Combettes and Pesquet in [7] , itself being an extension of the algorithmic 
scheme from [4] obtained by allowing also Lipschitzian monotone operators and parallel 
sums in the problem formulation. In the mentioned works, by means of a product space 
approach, the problem is reduced to the one of finding the zeros of the sum of a Lips- 
chitzian monotone operator with a maximally monotone operator. The latter is solved 
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by using an error-tolerant version of Tseng's algorithm which has forward-backward- 
forward characteristics and allows to access the monotone Lipschitzian operators via 
explicit forward steps, while set- valued maximally monotone operators are processed 
via their resolvents. A notable advantage of this method is given by both its highly 
parallelizable character, most of its steps could be executed independently, and by the 
fact that allows to process maximal monotone operators and linear bounded opera- 
tors separately, whenever they occur in the form of precompositions in the problem 
formulation. 

Before coming to the description of the problem formulation and of the algorithm 
from [7] , we introduce some preliminary notions and results which are needed throughout 
the paper. 

We are considering the real Hilbert spaces T~L and Qi, i = 1, . . . ,m, endowed with 
the inner product (•,•) and associated norm ||-|| = \J (•, •), for which we use the same 
notation, respectively, as there is no risk of confusion. The symbols — v and —> denote 
weak and strong convergence, respectively. By M++ we denote the set of strictly positive 
real numbers, while the indicator function of a set C C % is 5q '■ W — > M. := M. U {±00}, 
defined by 5c(x) = for x £ C and Sc(x) = +00, otherwise. For a function / : % — > M 
we denote by dom/ := {x £ % : f(x) < +00} its effective domain and call / proper if 
dom / 7^ and f{x) > — 00 for all x £ %. Let be 

r("H) := {/ : % — > M : / is proper, convex and lower semicontinuous}. 

The conjugate function of / is /* : % — > R, f*(p) = sup{(p, x) — f(x) : x £ 7-L} for 
all p £ H and, if / £ T('H), then /* £ T('H), as well. The (convex) subdifferential of 
/ : % ->• R at x £ H is the set <9/(x) = {p e 7i : f{y) - f{x) > (p,y- x) Vy £ ?£}, if 
fix) £ R, and is taken to be the empty set, otherwise. For a linear continuous operator 
Li : % — >• ^j, the operator L* : Qi ^ H, defined via (LiX,y) = (x,L*y) for all a; £ H 
and all y £ Qi, denotes its adjoint operator, for i £ {1, ... , m}. 

Having two functions /, g : 7^ — > R, their infimal convolution is defined by / □ g : 
■H — >■ R, (/np)(x) = inf^g-^ {/(y) + 5(2; — y)} for all x £ H, being a convex function 
when / and g are convex. 

Let M : % — > 2^ be a set-valued operator. We denote by gra M = {(x, u) £ % x % : 
n £ Mx} its graph and by ran M = {u £ H : 3x £ T~L, u £ Mx} its range. The inverse 
operator of M is defined as M^ 1 : % — > 2^, M _1 (u) = {x £ H : -u £ Mx}. The operator 
M is called monotone if (x — y, it — u) > for all (x, u), (y, £ gra M and it is called 
maximally monotone if there exists no monotone operator M' : % — > 2^ such that 
graM' properly contains graM . The operator M is called p-strongly monotone, for 
p £ if M — pld is monotone, i. e. (x — y,u — v) > p\\x — y\\ 2 for all (x, it), (y, i>) £ 

graM, where Id denotes the identity on %. The operator M : T~L — > % is called v- 
Lipschitzian for v £ M++ if it is single- valued and it fulfills \\Mx — My\\ < v\\x — y\\ for 
all x, y £ %. 

The resolvent of a set-valued operator M : % — > 2 n is Jm '■ T~i — > 2^ , Jm = 
(Id + M) _1 . When M is maximally monotone, the resolvent is a single- valued, 
1-Lipschitzian and maximal monotone operator. Moreover, when / £ r('H) and 
7 £ 5(7/) is maximally monotone (cf. [9, Theorem 3.2.8]) and it holds J^qj = 

(Id + 7<9/) ~ 1 = Prox 7 j. Here, Prox 7 j(x) denotes the proximal point of parameter 7 of 
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/ at x £ H and it represents the unique optimal solution of the optimization problem 

inf {/(y) + ^\\y -A?)- (1-1) 

For a nonempty, convex and closed set C C ^ and 7 £ K++ we have Prox 7< 5 c = Pes 
where : % — >■ C, Vc(%) = a r g mm zec ll x ~~ denotes the projection operator on C. 
Finally, the parallel sum of two set- valued operators Mi, M2 '■ H — > 2^ is defined as 

Mi □ M 2 : % -> 2 n , Mi □ M 2 = (M{ 1 + M^ ~ ' 



We can formulate now the monotone inclusion problem which we investigate in this 
paper (see [7]). 

Problem 1.1. Consider the real Hilbert spaces H and ft,i = l,...,m, A : ~H — > 2^ a 
maximally monotone operator and C:H->^a monotone and //-Lipschitzian operator 
for some fx £ Furthermore, let z £ H and for every % £ {1, . . . , m}, let rj £ Qi, 

let Bi : ft — > 2^* be maximally monotone operators, let Di : Qi — >■ 2^* be monotone 
operators such that D~ l is i/j-Lipschitzian for some Ui £ and let Li : Ti —> Qi be 

a nonzero linear continuous operator. The problem is to solve the primal inclusion 

m 

find x £ H such that z £ Ax + ^L* Di)(Ux - n)) + Cx, (1.2) 

i=i 

together with the dual inclusion 

find ^ e ft , . . . ,v m £ Q m such that (3x £ U) { _J " ( | ^ ^ ^ + J ! 1? _ ? m _ 

(1.3) 

Throughout this paper we denote by <? := ft x ... x £/ m the Hilbert space equipped 
with the inner product 

m 

((p 1} . . . ,p m ),(gi, • • -,q m )} = J2(pi,qi) V(pi, . . . ,p TO ) V(gi, ...,? m )G6 

i=l 

and the associated norm ||(pi, . . . ,p m )|| = VYa=i \\Pi\\ 2 f° r an (Pi> • • • >Pm) £ £?• We 
introduce also the nonzero linear continuous operator L : H — > Q, Lx = (L\x, . . . , L m x), 
its adjoint being L* : Q -> H, L*v = YT=i L *Vi- 

We say that (x,v\, . . . ,v m ) £ % x Q is a primal-dual solution to Problem 1.1, if 

m 

z — y~] L*Vi £ Ax + Cx and V{ £ (Bi □ Di)(Li~x — rj), % = 1, . . . , m. (1-4) 
i=i 

If (x,v\, . . . ,v m ) £ H x Q is a primal-dual solution to Problem 1.1, then x is a solution 
to (1.2) and iv\, . . . ,v m ) is a solution to (1.3). Notice also that 

m 

x solves (1.2) <^ z - ^L*(BinDi)(LiX - ri) £ Ax + Cx 44> 

i=l 

' z - ££1 L *iVi £ Ax + Cx, 



3 ui S ft , . . . , v m £ Q m such that 



Vi £ (BiUDi)(LiX - ri), i = l,...,m. 
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Thus, if x is a solution to (1.2), then there exists (vi,...,v m ) G Q such that 
(x,v\,. .. ,v m ) is a primal-dual solution to Problem 1.1 and if (vi,... ,v m ) is a solu- 
tion to (1.3), then there exists x G H such that (x,vi, . . . ,v m ) is a primal-dual solution 
to Problem 1.1. 

The next result provides the error-free variant of the primal-dual algorithm in [7] 
and the corresponding convergence statements, as given in [7, Theorem 3.1]. 

Theorem 1.1. For Problem 1.1 suppose that 

zeran (^ + f^L* i {B i UD i )(L i --r i ) + C^ . 
Let xq G % and (i>i,o, • • • , w m ,o) G Q, set 



(3 = max{/i, z^i . . . , v m } + 



\ 



£l|£< 



1=1 



choose e G (0, g^j) and {^ n )n>o a sequence in 



e, 



1-e 



and set 



(Vn > 0) 



For i 

Xn+l = 7 



(X n - 7n (C^n + Ei=l £*«i,n ~ z)) 



1, . . . , m 



JfnBr 1 ( V i,n + 7n(LiX r 



D 



-1 



— r. 



= 7„Li(pi,„ - 2„) + J n (Dt 1 V i , n - D^ 1 p 2 ,i ! n) +P2 
YT=1 L i( v i,n ~ P2,i,n) + 7n(Cx n - Cj>l,„) + p 1 



(1.5) 



,t,Tl 



T/ien the following statements are true: 

ft) EneN \\ x n ~ Pi,n\\ 2 < +°o and Vi G {1, . . . , m} EnsN IK« ~ P2,i,n|| 2 < +oo. 
(ii) There exists a primal-dual solution (x,vi, . . . ,v m ) £ H x Q to Problem 1.1 such 
that the following hold: 

(a) x n — 1 x and p\^ n — 1 x. 

(b) (Vi G {1, . . . ,m}) u i)fl Vj andp 2 ,i,n 

In this paper we consider first Problem 1.1 in its particular formulation as a primal- 
dual pair of convex minimization problems, approach which relies on the fact that 
the subdifferential of a proper, convex and lower semicontinuous function is maximally 
monotone, and show that the convergence rate of the sequence of objective function 
values on the iterates generated by (1.5) is of O(^), where n G N is the number of 
passed iterations. Further, in Section 3, we provide for the general monotone inclusion 
problem, as given in Problem 1.1, two new acceleration schemes which generate un- 
der strong monotonicity assumptions sequences of primal and/or dual iterates converge 
with improved convergence properties. The feasibility of the proposed methods is ex- 
plicitly shown in Section 4 by means of numerical experiments in the context of solving 
image denoising, image deblurring and image inpainting problems via total variation 
regularization. 
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One of the iterative schemes to which we compare our algorithms is a primal-dual 
splitting method for solving highly structured monotone inclusions, as well, and it was 
provided by Vu in [8]. Here, instead of monotone Lipschitzian operators, cocoercive op- 
erators were used and, consequently, instead of Tseng's splitting, the forward-backward 
splitting method has been used. The primal-dual method due to Chambolle and Pock 
described in [6, Algorithm 1] is a particular instance of Vu's algorithm. 



2 Convex minimization problems 

The aim of this section is to provide a rate of convergence for the sequence of the values 
of the objective function at the iterates generated by the algorithm (1.5) when solving 
a convex minimization problem and its conjugate dual. The primal-dual pair under 
investigation is described in the following. 

Problem 2.1. Consider the real Hilbert spaces H. and Gi,i = l,...,m, f G r(%) and 
li:W->Ra convex and differentiable function with ^-Lipschitzian gradient for some 
/i G Furthermore, let z G H and for every i G {1, . . . , m}, let r, G Gi, g%,l% G T(Gi) 

such that li is f~ 1 -strongly convex for some z/j G and let L, : H. — > Gi be a nonzero 

linear continuous operator. We consider the convex minimization problem 

(P) inf \f(x) + f^(g i nl i )(L i x-r i ) + h(x)-(x,z)\ (2.1) 
x€H { i=i J 

and its dual problem 

{/ m \ m ~| 

- (f □ />*) U - £ L* Vl - (gUvi) + lUvi) + (vi, n)) . 
\ i=l / i=l J 

(2.2) 

In order to investigate the primal-dual pair (2.1)-(2.2) in the context of Problem 
2.1, one has to take 

A = df, C = V/i, and, for i = 1, . . . , m, Bi = dgi and Di = dl{. 

Then A and Bi,i = 1, ...,m are maximal monotone, C is monotone, by [1, Proposition 
17.10], and D^ 1 = V7* is monotone and i/j-Lipschitz continuous for i = l,...,m, 
according to [1, Proposition 17.10, Theorem 18.15 and Corollary 16.24]. One can easily 
see that (see, for instance, [7, Theorem 4.2]) whenever (x, v\ , . . . , v m ) G % x Q is a 
primal-dual solution to Problem 1.1, with the above choice of the involved operators, x 
is an optimal solution to (P), (vi, . . . , v m ) is an optimal solution to (D) and for (P)-(D) 
strong duality holds, thus the optimal objective values of the two problems coincide. 

The primal-dual pair in Problem 2.1 captures various different types of optimization 
problems. One such particular instance is formulated as follows and we refer for more 
examples to [7]. 

Example 2.1. In Problem 2.1 take z = 0, let li : Gi —> K, h = 5{o} an d ?l = for 
i = 1, . . . , m, and set h : % — > R, h{x) = for all x G %. Then (2.1) reduces to 

{m 
/(*) + 5>(L 
i=i 
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while the dual problem (2.2) becomes 

{/ m \ m } 

-rl-^LtvA-^ivA. 
\ i=l / i=l J 

In order to simplify the upcoming formulations and calculations we introduce the 
following more compact notations. With respect to Problem 2.1, let F : % — > R, F(x) = 
f(x) + h(x) — (x, z). Then domP = dom / and its conjugate F* : H — > K is given by 
F*(p) = + + = (/*D/i*)(z + p), since domh = H. Further, we set 

v = (v 1 , . . . ,v m ), v = (vi,...,v m ), p 2 ,n = (j>2,l,n,---iP2,m,n), r = (r 1 , . . . , r m ) . 

We define the function G : Q — >• M, = Z^iGft ^ h)(yi) and observe that its 

conjugate G* : 1 is given by G*(v) = TZM^kY^) = TZM + **)«>■ 
Notice that, as l*,i = 1, . . . ,m, has full domain (cf. [1, Theorem 18.15]), we get 

dom G* = (dom n dom I*) x . . . x (dom g* m n dom/^J = dom 5^ x . . . x dom 5^, 

(2-3) 

The primal and the dual optimization problems given in Problem 2.1 can be equiv- 
alently represented as 

(P) inf {F{x) + G{Lx-r)}, 
xeH 

and, respectively, 

(D) sup{-F*(-L*v) - G*(v) - (v,r)}. 
veg 

Then x G % solves (P), v G Q solves (D) and for (P)-(P) strong duality holds if and 
only if (cf. [2,3]) 

-L*v G dF(x) and Lx-r£ dG*(v). (2.4) 

Let us mention also that for x G % and v G Q fulfilling (2.4) it holds 

[(Lx -r,v) + F(x) - G*(v)] - [(Lx -r,v) + F(x) - G*(v)] > Vx G U Vw G Q. 

For given sets B\ C and B2 C 5 we introduce the so-called primal-dual gap 
function 

Gb 1 xB 2 {%,v) = sup {(isc-r,*) + P(x) - G*(v)} 

- inf {(Lx-r,v) + F(x)-G*(v)}. (2.5) 

We consider the following algorithm for solving (P)-(P), which differs from the one 
given in Theorem 1.1 by the fact that we are asking the sequence (7 n )n>o Q ^++ to be 
nondecreasing. 
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Algorithm 2.1. Let xo £ H and (ui,o, • • • , w m ,o) £ set 

n 

(3 = max{/i, z/i, . . . , u m } + T ||-M 2 ; 

\ i=i 

choose e G ^0, and (7 n ) n >o a nondecreasing sequence in 



e i=s 



and set 



(Vn > 0) 



pi,„ = Prox 7n/ (x n - 7 n (V/i(x„) + ^>i,n - z)) 
For i = 1, . . . ,m 

P2,i,n = Pr OX 7ng * (fj,„ + 7n(LjX n ~ Vl*{v i)n ) - r;)) 
Vi, n +1 = lnLi{pi, n ~ X n ) + 7n(VZ*(fi,„) - VZ*(j? 2 ,i,n)) + P2,t,n 
X n +1 = 7nEELl L ii v i,n ~P2,i,n) + 7n(V/i(x„) - V/t(pi, n )) +W,n- 



(2.6) 



Theorem 2.1. For Problem 2.1 suppose that 

z £ ran (d/ + £ L| (% □ 5/,) (L, • -r<) + . 

T/ien there exists an optimal solution x £ H to (P) and an optimal solution 
(vi, . . . ,v m ) G Q to (D), such that the following holds for the sequences generated by 
Algorithm 2.1: 

(a) z - YT=\ e df{x) + Vh{x) and Ux - n e dg*{vi) + VZ*(^) Vi € {1, . . . , m}. 

(b) x n -± x, pi )Tl x and v i)n «j, p 2 ,i,n «i Vi € {1, . . . ,m}. 
(cj For n > it holds 

ll — ll2 m ii — n2 n — n2 m ii — n2 

lx„. -x|r ^ ||vi in - Vi\\" \\xq - x|| z ,^11^0-^1 



2 7 « 



+ E 

i=l 



2 7 « 



270 



+ E 



270 



i=l 

.A ._ 1 s^N-l 



(d) If B\ C ?i and B 2 C C/ are bounded, then for x := y^ n=0 pi jW and : = 



1 J2n=o P2,i,n, i = 1, ■ ■ ■ ,m, the primal-dual gap has the upper bound 



Gb 1 xB 2 (x ,v 1 ,...,v m )< — . 



(2.7) 



where 



C{B X ,B 2 ) 



sup 



\\2 m II 112 ' 

■<•(> - -'11 + ^ IKv.n - <vlr 



(x,v 1 ,...,v m )€B 1 xB 2 { 2 7o . =1 27o J 

(ej TTie sequence (x N , v x , . . . , v^) converges weakly to (x, v±, . . . , v m ). 

Proof. Theorem 4.2 in [7] guarantees the existence of an optimal solution x £ T~L to 
(2.1) and of an optimal solution (y\, . . . , v m ) £ C? to (2.2) such that strong duality holds, 
x n — 1 x, pi jn — x, as well as »j )n — and P2,i,n — ^ v% for i = 1, . . . , m, when n converges 
to +oo. Hence (a) and (b) are true. Thus, the solutions x and v = (v±, . . . ,v m ) fulfill 
(2.4). 
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Regarding the sequences (pi, n )n>0 and (p2,j,n)n>0; i = 1, . . . ,m, generated in Algo- 
rithm 2.1 we have for every n > 

Pi jTl = (Id + 7„3/) _1 (x n - 7„ (V/i(x n ) + L*u„ - z)) 

& Xn ~ Pl ' n - Vh(x n ) - L*v n + z G df( Pl , n ) 

In 

and, for i = 1, m, 

PZ,i,n = (Id + lndg*)~ (Vi,n + ln{LiX n - S7l*(v i>n ) - n)) 

yv v i,n P2,i,n , T wr*/ \ ^ a */ \ 

h LiX n - Vli [v i>n ) - r; G dg i {P2,i,n)- 

In 

In other words, it holds for every n > 

f(x) > /(pi,„) + ^ an ~ Pl - n - V/i(x n ) - L*t;„ + z, x - p 1>n ^ Vx G H (2.8) 
and, for i = 1, . . . , m, 

9*{vi) > 9*{P2,i,n) + (— — +LiX n -Vlt{v in ) -n,Vi -P2±n) VUj G Qi. (2.9) 
\ 7n / 

In addition to that, using that h and = 1, ...,m, are convex and differentiable, it 
holds for every n > 

%) > %i,„) + (V%i, n ), x - pi,„) Vx G ft (2.10) 

and, for i = 1, . . . , m, 

l*( v i) - l i(P2,i,n) + (Vl*(p2,i,n),Vi ~ P 2,i,n) VUj G £j. (2.11) 

Consider arbitrary x £ T~L and i? = . . . , u m ) G £/. Since 

\ II l|2 || ||2 II ||2 

Xn Pl.n \ W^n Pl,n , X pi n \\ \\%n X 

,X — pi n ) = 1 

In ' I 2jn 27„ 27„ 

Vi, n -P2,i,n \ \\Vi,n ~ P2,i,n\\ 2 . IK ~ P2,i,n\\ 2 \\Vi,n ~ Vi\\ 2 . 

,Vi-p2,i,n ) = 7. 1 7, n = 1 ,-, m , 

7n / 27 n 27„ 27 n 

we obtain for every n > 0, by using the more compact notation of the elements in Q 
and by summing up the inequalities (2.8)-(2.11), 

\\x n - X|| 2 \\v n ~ HP > \\Xn-Pl,n\\ 2 \\x ~ Pl,n\\ 2 II v n ~ P 2 ,n II 2 \\ v ~ P 2 ,n\\ 2 



27n 27 n 27 n 27 n 27„ 27^ 

m m 

^2(LiX n + Vl*(p2,i,n) ~ Vl*( v i,n) ~ - P2,i,n) ~ ^{g* + l i)( v i) + (/ + h )(Pl,n) 

i=l i=l 

m 

+ (Vfc(pi,„) - V/i(x n ) - L*v n + z, x - pi >n ) - ^ + /*)(P2,i,n) + (/ + /»)(a 
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Further, using again the update rules in Algorithm 2.1 and the equations 

'Pl,n ~ Xn+l 



In 

and, for i = 1, m 

' ' P2,i,n — Vi : n+\ 



,x-p ln 



||x n+ i-x|| 2 \\x n+ i -pi, n \\ 2 ||x-pi jn || 2 



2j„ 



2 7 n 



7n 



, Vi - P2,i, n 



\v it n +1 - Vi\\ 2 |Kn+l ~P2,i,n\\ 2 II V i ~ V%i,n || 2 



2 7 „ 2 7i 

we obtain for every n > 

||x n -x|| 2 ||u„-u|| 2 ||x n+ i-x|| 2 - v|| 2 



2 7 n 



2 7n 



+ 



> 



2 7ri ~ 2 7n 2 7 

\\x n+1 -Pl,n\\ 2 _ IK+1 -P 2 ,n " 2 
2 7 n 2 7n 

^ - r,pa n ) - G*(p 2jn ) + F(s) 



, Jan -Pl.nll 2 IK -P 2 ,nl! 2 

2"„ ! 2 7n 
+ [(L Pl>n -r,v)-G*(v) + F( Pl>n )] 

(2.12) 



Further, we equip the Hilbert space 1-L = H x C/ with the inner product 
((l/,p),(*,g)) = (y,z) + (p,q) V(y,p),(z,g) G ft x Q 



(2.13) 



and the associated norm ||(y,p)|| = \/||y|| 2 + ||p|| 2 for every (y,p) G 7i x Q. For every 
n > it holds 

\\x n+1 - p hn \\ 2 , -P2,nl! 2 ||(»n+l,«n+l) ~ (Pl,n,P 2li ^' L ' 



2 7 „ 



+ 



2 7 n 



2 7n 



and, consequently, by making use of the Lipschitz continuity of Vh and Vi*, i 
1, . . . , m, it shows that 

\\(x n+1 ,v n+ i) - (pi 

,ri)P2,n) II 

= j n \\(L*(v n - p 2n ),L 1 (pi )n -x n ),.. .,L m (pi >n - X n )) 

+ (Vh(x n ) - Vh(pi >n ), Vi*(wi, n ) - Vi*(p 2 ,l,n), • ■ • , Vi;(%, n ) - Vi*(p 2 ,m,n)) 

< 7n || (£*(■?;„ - p 2j „),Li(pi )n -»„),.. . ,L m (pi,n - a?„))|| 

+ 7n ||(V/t(0 - V/l(pi,„), VZI(U1,„) - VII(p2,l,n), • • • , V4(%,„) - Vi*0 2) m,n))|| 



7r. 



\ 



^L*{v^ n -P2,i,n) 



i=l 



+ J2\\ L i(Pl,n ~ XnW 



i=l 



In 



\ 



\\Vh{x n ) - V/l(pi, n )|| 2 + J2 II Vi*Kn) " Vi*(p 2)i , n ) 



< 



7n 



P2,i,n|| 2 + X! ll Li H 2 l^ 1 '" ~ Xn \ 



i=l 



\i=l 



+ lr, 



\ 



H 2 \\x n -Pl,n|| 2 + ^2^i\\Vi,n ~ P2,i,n\\ 2 
i=l 
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< 



In 



*i E W Li W 2 +max{/i,i^ 3 . . . ,i/ m } ] ||(a;„,u n ) - (pi,n,P2,n)ll- 



(2.14) 



Hence, by taking into consideration the way in which (7 n )n>o is chosen, we have for 
every n > 

1 



27r, 



\x n -pi,n\\ 2 + \\Vn -P2,nl! 2 ~ \\x n +l -£>l,n|| 2 ~ ||«n+l ~ f»2.nl| 2 



> 



2 7 n 



1-7 2 



^ E ||Lj|| 2 + max{/u,z/i, . . . ,i/ m } ] ] ||(x,v n ) - (Pi,n,P 2 ,JI| 2 > °- 



and, consequently, (2.12) reduces to 



I - |2 II ||2 || ||2 

H ~ > ~ 1- [(-kPl.n - r,v) - G (v) + F(pi, n )] 



27, 



27n 



7n 



27 



n+l 



+ 



7n+l - «| 



Lx - r,p 2 ) - G*(p 2 J + F(x) 



In 27 n+ i 

Let N > 1 be an arbitrary natural number. Summing the above inequality up from 
n = to N — 1 and using the fact that (7«) n >o is nondecreasing, it follows that 

II 112 II 112 II 112 N—l 

\\x -x\\ A \\Vq — v\\ \\xn — x\\ ^ 

—2^— + —2^— > 27jV + E t(^l.n ~r,v) - G (v) + F( Phn )} 



\vn — v\ 

2 7 iv 



n=0 
2 JV-1 

" " E [(^ " ^n) " G*(p 2in ) + F(x) 



n=0 



(2.15) 



Replacing x = x and t; = v in the above estimate, since they fulfill (2.4), we obtain 

N-l N-l 



J2 [(Lp ljn -r,v)-G*(v) + F( Pl , n )}-J2 [(Lx-r,p 2in )-G*(p 2jn ) + F( 

n=0 n=0 

Consequently, 

||xq — x\\ 2 \\vo — v\\ 2 ^ ||xjv — x\\ 2 \\vn — v\\ 2 



X 



> 0. 



2 7 o 270 27at 27at 

and statement (c) follows. On the other hand, dividing (2.15) by N, using the convexity 



of F and G* , and denoting x N := 4 J2n=o Pi,n and := 4 J2n=o P2,i,n, i = 1, . . . , m 



we obtain 



1 / llxo — a;|| 2 ||vo — v l| 2 



iV \ 2 70 



+ 



270 



> 



Lx N - r,vj - G*(v) + F(x N 
Lx - r,v N ) - G*(v N ) + F(x 



which shows (2.7) when passing to the supremum over x E B\ and v 6 B 2 . In this way 
statement (d) is verified. The weak convergence of (x N ,v N ) to (x,v) when N converges 
to +oo is an easy consequence of the Stolz-Cesaro Theorem, fact which shows (e). □ 
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Remark 2.1. In the situation when the functions gi are Lipschitz continuous on Qi, i = 
1, m, inequality (2.7) provides for the sequence of the values of the objective of (P) 
taken at (x n )n>i a convergence rate of O(^), namely, it holds 

F(x N ) + G(Lx N - r) - F(x) - G(Lx - r) < 9}^M y N > L ( 2 .16) 

Indeed, due to statement (b) of the previous theorem, the sequence (pi, n )n>0 Q H is 
bounded and one can take B% C H being a bounded, convex and closed set containing 
this sequence. Obviously, x E B%. On the other hand, we take Bi = dom g* x . . . x 
domgj^, which is in this situation a bounded set. Then it holds, using the Fenchel- 
Moreau Theorem and the Young- Fenchel inequality, that 

g Bl xB 2 (x N ,v N )=F(x N ) + G(Lx N -r) + G*(v N )- inf {(Lx - r, v N ) + F{x)\ 

> F(x N ) + G(Lx N -r) + G*(v N ) - (Lx - r, v N \ - F(x) 

> F(x N ) + G(Lx N - r) - F(x) - G{Lx - r). 

Hence, (2.16) follows by statement (d) in Theorem 2.1. 

In a similar way, one can show that, whenever / is Lipschitz continuous, (2.7) 
provides for the sequence of the values of the objective of (D) taken at (u )jv>i a 
convergence rate of O(jr). 

Remark 2.2. If Qi, i = 1, . . . , m, are finite- dimensional real Hilbert spaces, then (2.16) 
is true, even under the weaker assumption that the convex functions gi,i = l,...,m, 
have full domain, without necessarily being Lipschitz continuous. The set B\ C T~L can 
be chosen as in Remark 2.1, but this time we take B2 = X^Li Un>o (-^iPl,n) C G, by 
noticing also that the functions gi,i = 1, ...,m, are everywhere subdifferentiable. 

The set B2 is bounded, as for every i = 1, . . . , m the set Un>o ®9i {LiPi,n) is bounded. 
Let be i G {l,...,m} fixed. Indeed, as p\ jn x, it follows that Lip\^ n — > LiX for 
i = 1, ...,m. Using the fact that the subdifferential of gi is a locally bounded operator 
at LiX, the boundedness of Un>o (LiPi,n) follows automatically. 

For this choice of the sets B\ and B2, by using the same arguments as in the previous 
remark, it follows that (2.16) is true. 

3 Zeros of sums of monotone operators 

In this section we turn our attention to the primal-dual monotone inclusion problems 
formulated in Problem 1.1 with the aim to provide accelerations of the iterative method 
described in Theorem 1.1 under the additional strong monotonicity assumptions. 

3.1 The case when A + C is strongly monotone 

We focus first on the case when A + C is p-strongly monotone for some p E K ++ and 
investigate the impact of this assumption on the convergence rate of the sequence of 
primal iterates. The condition A + C is p-strongly monotone is fulfilled when either 
A : Ji — > 2 or C : T-L — > Ji is p-strongly monotone. In case that A is pi-monotone and 
C is /^-monotone, the sum A + C is /9-monotone with p = p\ + P2- 
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Remark 3.1. The situation when B^~ + D^ 1 is Tj-strongly monotone with Tj G K++ 
for i = 1, . . . , m, which improves the convergence rate of the sequence of dual iterates, 
can be handled with appropriate modifications. 

Due to technical reasons we assume in the following that the operators D^ 1 in 
Problem 1.1 are zero for i = 1, . . . , m, thus, -D;(0) = Qi and Di{x) = for i / 0, for 
i = 1, ...,m. In Remark 3.2 we show how the results given in this particular context 
can be employed when treating the primal-dual pair of monotone inclusions (1.2)-(1.3). 
Consequently, the problem we deal with in this subsection is as follows. 

Problem 3.1. Consider the real Hilbert spaces H and Qi,i = l,...,m, A : % — > 2^ a 
maximally monotone operator and C:H->^a monotone and //-Lipschitzian operator 
for some \i G Furthermore, let z G H and for every % G {1, . . . , m}, let rj G Qi, 

let Bi : Qi — >■ be maximally monotone operators and let Lj : % — > Qi be a nonzero 
linear continuous operator. The problem is to solve the primal inclusion 

m 

find x G H such that z G Ax + 2_, L*Bi(LiX — rj) + Cx, (3.1) 

i=i 

together with the dual inclusion 



find v\ G Qi, . . . , v m G Qm such that (3x G Wj 



z ~ E™ i L *iVi eAx + Cx 

Vi G Bi(LiX - rj), i = l,...,m. 



(3.2) 

The subsequent algorithm represents an accelerated version of the one given in 
Theorem 1 . 1 and relies on the fruitful idea of using a second sequence of variable step 
length parameters (o~ n ) n >o Q which, together with the sequence of parameters 

(ln)n>o Q K-H-, P ia y an important role in the convergence analysis. 

Algorithm 3.1. Let xq G %, ( y i,0; ■ ■ 



70 G 0,min <^ 1, 




(Vn > 0) 



(3.3) 



2(l + 2pVj; u 2 70 (l + 2 /O )E^ill^|| 2 ' 

Consider the following updates: 

Pl,n = J ln A (%n - In (Cx n + YhL\ L*Vi,n ~ z)) 

For i = 1 , . . . , m 

P2,i,n = J a n BT l ( V i,n + °"n(-^i^n ~ n)) 
Vi,n+l = <7nLi(Pl,n ~ X n ) + P2,i,n 
X n +1 = In YT=1 L i( v i,n ~ P2,i,n) + ln{Cx n - Cp^ n ) + p 1>n 
On = 1/V 1 + 2 P7n(l - 7n), 7n+l = #n7n, 0"n+l = 0"n/#n- 

Theorem 3.1. In Problem 3.1 suppose that A + C is p-strongly monotone with p G M ++ 
and let (x, v\ , . . . , v m ) G H x Q be a primal-dual solution to Problem 3.1. Then for every 
n > it holds 

K-*|| 2 + 7n£^^ (3.4) 

where 7„, a n G x n G % and (v i >n , . . . , v m<n ) G are the iterates generated by 

Algorithm 3.1. 
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Proof. Taking into account the definitions of the resolvents occurring in Algorithm 3.1 
we obtain 



— - Cx n - V L*Vi. n + z G Ap 

In ^ 



l.n 



and m i=l 

Vi.n P2,i,n 



+ Ux n -r i ^B i 1 p 2 ,i, n , i = l,...,m, 



0~ n 

which, in the light of the updating rules in (3.3), furnishes for every n > 

m 

•En 



J2 L*P2,i,n + Z <E (A + C) V l,r. 

i= 

v i n ~ Vi n -\-l r>—l ■ 1 

— h LiPl,n ~ n G -B^ p 2 ,i,n,« = 1, • • • 



and 7n <=1 (3.5) 



The primal-dual solution (x, «i, . . . , v m ) £ HxQ to Problem 3.1 fulfills (see (1.4), where 
D^ 1 are taken to be zero for i = 1, m) 



z — L*v-i G Ax + Cx and Vi G B{(LiX — rj), i = 1, . . . , m. 
Since the sum ^4 + C is /^-strongly monotone, we have for every n > 



m / rra 

Xn -Eji+l 



( Pl,n - x, — L*P2,i.n + z - \z - ^ ) > /o||Pl,n - ^|| 2 (3.6) 

\ 7n i=l V 8=1 // 

while, due to the monotonicity of -B" 1 : Gi —> 2^, we obtain for every n > 

^P2,i,n - «i, J hn+1 + Lj_pi in - rj - (Ljx - r*)\ > 0, i = 1, ... ,m. (3.7) 
Further, we set 

V = (Vi, . . . ,V m ), V n = (vi tTl , ...,V min ), P 2 , n = (P2,l,n, ■ ■ ■ ,P2,m,n)- 

Summing up the inequalities (3.6) and (3.7), it follows that 

_ X n -X n+ l\ I _ V n -V n+1 \ I _ \ 

Pl,n ~ X, J + (p 2 ,n ~ V, J + (pi,n ~X,L (V - p 2 ,n) / 

+ (P2,n ~ V, L(j) 1>n - x)^ > p\\pi, n ~ x\\ 2 . (3.8) 

and, from here, 



Pl,n~X, ) + \P2n- v i ) > p\\Pl,n ~ x\\ Vn > 0. (3.9) 

In I \ &n I 

In the light of the equations 

Xn X n -\-\ \ / X n X n J,-l \ I X n X n -\-\ 

Pl,n ~ X, ) = < pi n - X n+ l, ) + < X n+ i - X, 



In I \ In I \ 7 

I i|2 II \\2 || ||2 || ||2 

|*£f»+l Pi, n|| \\Xn Pi, rail \\Xn X\\ \\X n -^i X\\ 



27„ 27„ 27„ 27 n 
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and 



P2,n ~ »> 



V n ~ Vn+1 
On 



P2,n ~ v n+l 



IK+1 ~ P2, 

2a n 



<?n I 



+ < V n+ i - 17, 



Vn ~ Vn+1 
&n 



\Vn ~ P2,n\\ 2 _ \\v n ~ v\\ 2 \\v n+1 - v\\ 2 



2a r , 



2a r , 



2a v, 



inequality (3.9) reads for every n > 



\x n - x\\ 2 \\v n - v\\ 2 



2a r , 



II i|2 || n2 II i|2 

. II — n2 , P^n+l — X\\ \\Vn+l ~ V\\ \\ x n ~ Pl,n\\ 

>P\\Pl,n-X\\ H 1 h 



27„ 2a n 2j n 

\Vn-P2,n\\ 2 \\x n +l ~ Pl,nf IK+1 - P 2 , " L> 



2<T„, 



2 7 n 



2o>, 



. (3.10) 



Using that 2ab < aa 2 + — for all a, & £ M, a £ we obtain for a := 7 n , 

Plbl.n - ^|| 2 > P (lkn+1 - S|| 2 - 2\\x n+ i - x\\\\x n+ i - Pl iTl || + ||x n+ l 

2p 7n (l-7„) 2 2^(1 -7ft).. | 2 

> 7, \\X n+ l-pi >n \\ , 

2 7n 2 7n 

which, in combination with (3.10), yields for every n > 
'\x n -x\\ 2 \\v n -v\\ 2 (1 + 2/3 7n (l - 7n ))||x n+ i -x\\ 2 \\v n+1 -v\\ 2 



+ 



> 



+ 



27n 2cr n 2 7n 2cr n 

2 "«n -p 2 ,nl| 2 (1 +2p(l -7„))||x n+ i -Pl,„|| 2 ll^n+1 -P 2 ,nl| 2 



I ||3-n Pi, n|| 

27n 2cr n 



2 7 n 



2cr n 



(3.11) 



Investigating the last two terms in the right-hand side of the above estimate it shows 
for every n > that 



> 



> 



(1 + 2p(l - 7 n ))||x n+ i -pi, n \? 

27n 

(1 + 2p)7» 
2 

m 

2 



^2 L i(Vi,n ~ P2,i,n) + (Cx n - Cpi >n ) 



2(1 + 2p) ln 



E 

\i=l 



Vn -P2,n\\ 2 + P 2 ||^n ~ Pl.n 



and 



[K+i -P2„ 
2a n 



a 

"2 



l.n "^r 



> 



v.i=l 



E 

Vi=l 



||Pl,n - »n|| 2 - 
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Hence, for every n > it holds 

lkn-Pl.nl! 2 , \\ V n-P2,n\\ 2 (1 + 2p(l - 7„)) \\x n+1 - p 1>n || 2 ||« n+1 - P 2 ,J 



27n 2cr n 27„ 2a n 

. (1 ~ 7n*Xn S£l ll^ll 2 ~ 2(1 + 2p) 7 ^ 2 ) „ _ ||2 

— o Pl.n 

27n 

, (l-2 7nC T ri (l + 2p)E™ill^|| 2 ) l , „ „2 



2d r , 

> 0. 



\ v n ~ Pi, n \ 



The nonnegativity of the expression in the above relation follows because of the sequence 
(7n)n>o is nonincreasing, 7„cr n = 70 Co for every n > and 

/ f Jl + Ap \\ , 1 

7n 6 0, min < 1, — r — > and <7n = ; . — = — n — tttt- 

7 V 1 '2(l + 2p)/i/y 2 7 o(l + 2 /3 )E™ill^ll 2 

Consequently, inequality (3.11) becomes 

||x n -s|| 2 ||u„ -tf|| 2 (1 +2p7„(l -7 n ))||x n+ i -x\\ 2 \\v n+ i-v\\ 2 

H ~ > ~ 1 ~ Vn > L). 



27n 2<r n 27„ 2<r n 

(3.12) 

Notice that we have 7 n +i < 7m Cn+i > Cn and 7 n +iCn+i = 7n^"n for every n > 0. 
Dividing (3.12) by 7 n and making use of 

a 1 a an 

y n — , , . , 7n+l — Vnlm <T n +l = 

VI + 2p7 n (l - 7 n ) t/„ 



we obtain 



l^n — ^ll 2 ll^n — v\\ 2 \\x n+ i — x\\ 2 \\v n+ i — v\\ 2 , , 

ii + n_n y_ > ILiLbL n_ + ]LJ^ — Vn > 0. 



27„ 27„o- n 2 7 ^ +1 27 n+ icr n+ i 

Let be iV > 1. Summing this inequalities from n = to N — 1, we finally get 

IN) -^|| 2 IK - ^|| 2 > Ikjv -^|| 2 Ih^v -^|| 2 , 3 13 x 

27^ 27ocro ~ 27^ 2-/ N a N ' 1 "'' 

In conclusion, 

||rc n -x|| 2 ||u n -i>|| 2 2 /ll x o-^|| 2 . \\v -v u2 ^ 



2 2cr n y 275 27 cr y 

which completes the proof. □ 

Next we show that p^ n converges like - as n — > +00. 
Proposition 3.2. Let 70 G (0, 1) and consider the sequence (7 n )n>o Q ^++> where 

ln+i = ;1 ^" ^ Vn > 0. (3.15) 

VI + 2p7„(l - 7 n ) 

T/ien lim n ^ +OC) np7„ = 1. 
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Proof. Since the sequence (7 n )n>o Q (0, 1) is bounded and decreasing, it converges 
towards some I 6 [0, 1) as n — > +oo. We let n — > +oo in (3.15) and obtain 

l 2 (l + 2pl(l - I)) = I 2 & 2p/ 3 (l -0 = 0, 

which shows that I = 0, i.e. j n — > (n — > +oo). On the other hand, (3.15) implies that 
— > l(n — >■ +oo). As (^)n>o is a strictly increasing and unbounded sequence, by 
applying the Stolz-Cesaro Theorem it shows that 

n n + l-n 7«7n+i 
nm n7 n = nm —r- = nm — =- = nm 



H 7»7n+l(7n + 7n+l)( 3A5) 7n7n+l (7n + 7n+l) 

"^+°° 7n~7n+l n ^+°° 2p7^ +1 7„(l -In) 

r 7n + 7n+l ^TT + 1 2 1 

= lim = lim ■ = — = — 

n-*+oo 2p7 n+ i(l - 7 n ) n->+oc 2p(l - 7„) 2p p' 

which completes the proof. □ 

Hence, we have shown the following result. 

Theorem 3.3. In Problem 3.1 suppose that A + C is p-strongly monotone and let 
(x,vi, . . . ,v m ) £ T~L x Q be a primal-dual solution to Problem 3.1. Then, for any e > 0, 
there exists some n$ £ N (depending on e and p7oj such that for any n > uq 

i i / n — n2 m n — n2 \ 

where 7 n , a n £ x n £ H and (vx >n , . . . ,v m>n ) £ Q are the iterates generated by 

Algorithm 3.1. 

Remark 3.2. In Algorithm 3.1 and Theorem 3.3 we assumed that D~ l = for i = 
1, . . . ,m, however, similar statements can be also provided for Problem 1.1 under the 
additional assumption that the operators Di : Gi —> 2 Si are such that D~ is - 
cocoercive with Vi £ R++ for i = 1, . . . , m. This assumption is in general stronger than 
assuming that Di is monotone and I?" 1 is z/j-Lipschitzian for i = 1, ...,m. However, it 
guarantees that Di is f~ 1 -strongly monotone and maximally monotone for i = l,...,m 
(see [1, Example 20.28, Proposition 20.22 and Example 22.6]). We introduce the Hilbert 
space H = % x Q, the element z = (z,0,. . . ,0) £ H and the maximally monotone 
operator A : T-L — > 2 W , A(x, y±, . . . , y m ) = (Ax, D\y\, . . . , D m y m ) and the monotone and 
Lipschitzian operator C : H —¥ H, C(x, y%, . . . , y m ) = (Cx, 0, . . . , 0). Notice also that 
A + C is strongly monotone. Furthermore, we introduce the element r = (n, . . . ,r m ) £ 
Q, the maximally monotone operator B : Q — > 2 s , B(yi, . . . , y m ) = (Biyi, . . . , B m y m ), 
and the linear continuous operator L : H — > G, L(x, y± ■ ■ ■ , y m ) = (L\x — yi,..., L m x — 
y m ), having as adjoint L* : Q ->• 'H, L*(qi, . . . , q m ) = (Y%Li L*Qi, -qi, ~9m)- We 
consider the primal problem 

find x = (x,p 1 . . .p m ) £ ~H such that z £ Ax + L*B (Lx — r) + Cx, (3.17) 



16 



together with the dual inclusion problem 



find v G G such that (3x ^U) \ Z L *f r £ Ax , + ° X . (3.18) 
v M v G B(Lx -r) y ' 

We notice that Algorithm 3.1 can be employed for solving this primal-dual pair of 
monotone inclusion problems and, by separately involving the resolvents of A, Bi and 
Di , i = 1 , . . . , to, as for 7 G M ++ 

J-yA(x,yi, ■ ■ ■ ,y m ) = (J-yAX, J-yDxVu---, J-yDnVm) V(x,yi, . . . ,y m ) G U 
J y B(qi, ...,q m ) = (JyBiqi, • • • , J-yB m q m ) V(gi, . . . ,q m ) G Q. 

Having (x, v) G H x <? a primal-dual solution to the primal-dual pair of monotone 
inclusion problems (3.17)-(3.18), Algorithm 3.1 generates a sequence of primal iterates 
fulfilling (3.16) in H. Moreover, (x, v) is a a primal-dual solution to (3.17)-(3.18) if and 
only if 

z — L*v G Ax + Cx and v G B (Lx — r) 

in 

44> z — L*tJj G Ax + Cx and Vi G -Dift, G Bi (LiX — p 4 — rj) , i = 1, . . . , m 
i=i 

m 

<J4> z — G j4x + Cx and t>j G -Djpj, L(x — ri G B~ l Vi +p i ,i = l,...,m. 

i=i 

Thus, if (s, v) is a primal-dual solution to (3.17)-(3.18), then (x, v) is a primal-dual 
solution to Problem 1.1. Viceversa, if (x, v) is a primal-dual solution to Problem 1.1, 
then, choosing p { G D^ 1 Vi,i = 1, ...,m, and 2: = (x,^ . . .p m ), it yields that (x,v) is a 
primal-dual solution to (3.17)-(3.18). In conclusion, the first component of every primal 
iterate in H generated by Algorithm 3.1 for finding the primal-dual solution (x, v) to 
(3.17)-(3.18) will furnish a sequence of iterates verifying (3.16) in % for the primal-dual 
solution (x, v) to Problem 1.1. 

3.2 The case when A + C and B^ 1 + D~ x , i = l,...,m, are strongly 
monotone 

Within this subsection we consider the case when A-\-C is p-strongly monotone with p G 
and B~ 1 +D^~ 1 is Tj-strongly monotone with G M++ for i = 1, . . . , m, and provide 
an accelerated version of the algorithm in Theorem 1.1 which generates sequences of 
primal and dual iterates that converge to the primal-dual solution to Problem 1.1 with 
an improved rate of convergence. 

Algorithm 3.2. Let xq G H, (t>i,o> • • • ,Vm,o) £ and 7 G (0, 1) such that 

1 

7 < 7 v-. 

yjl + 2min{/9, n, . . . ,r m } \\ L i\\ 2 + max {^,1/1, . . . ,v m }) 
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Consider the following updates: 



(Vn > 0) 



Pl,n = J-yA (x„ ~ 7 (Cx n + Ya=1 L i v i,n ~ z)) 

For i = 1, . . . ,m 

P2,i,n = J^B- 1 ( v i,n + l{Li%n - D^V^n - Ti)j 
Vi, n +1 = lLi{pi^ n - X n ) + 7(Dr 1 Uj ) n - D^ 1 p 2 , i) n) + P2,i,: 
Xn+1 = 7EE=1 L*{Vi,n ~P2,i,n) + l{Cx n - Cpi n ) + Pl, n - 



(3.19) 



Theorem 3.4. In Problem 1.1 suppose that A + C is p-strongly monotone with p G 
R-f+, B^ 1 + D^ 1 is Ti-strongly monotone with Tj G /or i = l,...,m, and /ei 

. . . ,v m ) G % x C? be a primal-dual solution to Problem 1.1. Then for every n > 
it holds 



Xn X 



+ IK" ~~ ^ll 2 - ( 7 

i=l ^ 



1 



+ 2p min j(l - 7) 



\XQ - X\\ 2 + Y \\Vifi - Vi\ 
i=l 



where p m in = min{p, n, . . . ,r m } and x n £ H and (t>i,n, • • • G are i/te iterates 
generated by Algorithm 3.2. 

Proof. Taking into account the definitions of the resolvents occurring in Algorithm 3.2 
we obtain for every n > 



•Era 

7 



J] ^*P2,i,n + ze(A + C)pi, n 
i=l 



and 



7 



+ Ljpi jn - n G 1 + 1 )p2,i,n, i = 1, . . . , m. 



The primal-dual solution (x,v\, . . . ,v m ) G T~L x to Problem 1.1 fulfills (see (1.4)) 

m 

z — ^ L*Vi G + and G (i?j □ Di){Lix — r^), i = 1, . . . , m. 



i=i 



By the strong monotonicity of vl + C and B i 1 + D i , i = 1, . . . , m, we obtain for every 
n > 



Pl,ra - 26, 



and, respectively, 



%n+l 



7 



2 Ltp 2)i>n + z - ^ - £ L*Vij j > p\\ Phn - x\\ 2 (3.20) 



i=l 



— v i,n Vi,n+1 . T It — \\ \ II — II 2 • 1 

P2,i,Ti - Vi, h LiPi,n -n- (LiX -n))> Ti\\p2,i,n ~ Vi\\ ,1=1, ...,m. 

7 / 

(3.21) 

Consider the Hilbert space H = % x Q, equipped with the inner product defined in 
(2.13) and associated norm, and set 



— U 1) • • • j ^m) i "^n — i^ra ) ^ljTij • • • ) ^m,n) > Pn — \Pl,n , P2,l,n, • • • , P2,m,n)- 
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Summing up the inequalities (3.20) and (3.21) and using 

Pn ~ x , 



II 1 1 2 || 1 1 2 II 1 1 2 || 

•En 3?n+l \ H-En+1 Pn\\ \\Xn Pn\\ \\Xn X\\ ||3?n+l 



7 / 27 27 27 27 

we obtain for every n > 

II — 1|2 || — 1|2 || ||2 || ||2 

\\Xn ""II ^ || n2 . ""II ||*"n Pnll Pnll 

^ >Pmin||P n -^|| + ^ + ^ ^ ■ 

Further, using the estimate 2ab < 7a 2 + — for all a, b £ R, we obtain 

H _||2 ^ 2p min 7(l - 7) 2 2p min (l-7) _ 2 

27 27 

. 2p mi n7(l - 7) 2 2/3 min 2 

> n - 35 n «n+l ~ Pn Vn > 0. 

27 27 



Hence, (3.22) reduces to 



\x n - x\\ 2 (1 + 2p min 7(l - 7))||x n+ i - x\ 



27 ~ 27 

Ijgn -Pnll 2 _ (1 +2p min )||ir w+ i -p n \\ 2 > 

27 27 ' 

Using the same arguments as in (2.14), it is easy to check that for every n > 

\\ x n -Pn\\ 2 _ (1 + 2p min )||a; n+ i -p n \\ 2 
27 ~ 27 

/ 

1 - (1 + 2p min ) 7 2 



> 



V 



^ ||L;|| 2 +max{/i, i/i, . . . ,i/ m } 



I l|2 
I 33 " ~~ Pnll 



>o, 

whereby the nonnegativity of this term is ensured by the assumption that 

1 

7 < 7 v-. 

VI + 2p min (VEE^l ll-^ill 2 + max {A*, 1^1, ... , ^m}J 

Therefore, we obtain 

\\x n ~ x\\ 2 > (1 + 2p min 7(l - 7))||x n+1 - x\\ 2 Vn > 0, 

which leads to 

\\x n -x\\ 2 < I — — r ||sc - «|| 2 Vn > 0. 

VI + 2/3 min 7(l -7)/ 
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4 Numerical experiments in imaging 



In this section we test the feasibility of Algorithm 2.1 and of its accelerated version 
Algorithm 3.1 in the context of different problem formulations occurring in imaging and 
compare their performances to the ones of two other popular primal-dual algorithms 
introduced in [6]. For all applications discussed in this section the images have been 
normalized, in order to make their pixels range in the closed interval from (pure black) 
to 1 (pure white). 



4.1 TV-based image denoising 

Our first numerical experiment aims the solving of an image denoising problem via total 
variation regularization. More precisely, we deal with the convex optimization problem 



taf.{A7V(«) + 5l|.-H| ! '}. (4-1) 



where A € is the regularization parameter, TV : W 1 — >■ R is a discrete total 

variation functional and b G M n is the observed noisy image. 

In this context, x G W 1 represents the vectorized image X e R MxN , where n = M-N 
and Xij denotes the normalized value of the pixel located in the i-th row and the j-th. 
column, for i = 1, . . . , M and j = 1, . . . , N. Two popular choices for the discrete total 
variation functional are the isotropic total variation TV- iSO : W 1 — > M, 

M-l N-l 



TV iso (x) = V ( Xi+l j ~ Xi <^ 2 + ~ x m) 2 

i=i j=i 

M-l N-l 

+ X! \ X i+l,N — x i,N\ + X! \ X M,j+l — XM,j\ , 
i=l j=l 

and the anisotropic total variation TT4 n i so : M n — > M, 

M-l N-l 

TV SLn i so {x) — ^ ' ^ ' l^i+ljj x i,j\ \ x i,j+l x i t j\ 
i=l j=l 
M-l N-l 

+ 51 \ x i+h N ~ X %N\ + X! \ x M,j+l — XM,j \ , 
i=l j=l 

where in both cases reflexive (Neumann) boundary conditions are assumed. 

We denote y = E n x R n and define the linear operator L : W 1 — > y, Xij H> 
(LiXi i j,L 2 x ii j), where 



x i+ ij - x^, if i < M j x it j+i - Xij, if j < N 

0, Hi = M andjL 2^-< Q) ifj = JV 



The operator L represents a discretization of the gradient using reflexive (Neumann) 
boundary conditions and standard finite differences. One can easily check that ||L|| 2 < 8 
and that its adjoint L* : y — > ~SL n is as easy to implement as the operator itself (cf. [5]). 



20 





a = 0.12, A = 0.07 


a = 0.06, A = 0.035 


e = 10" 4 


£ = 10~ e 


£ = 10- 4 


£ = 10~ e 


ALGl 
ALG2 
PDl 
PD2 


350 (7.03 s) 
101 (2.28 s) 
342 (3.59 s) 
96 (1.02 s) 


2989 (59.82 s) 
442 (9.91 s) 
3133 (32.68 s) 
442 (4.67 s) 


184 (3.69 s) 
72 (1.62 s) 
180 (1.91 s) 
69 (0.76 s) 


1454 (29.07 s) 
298 (6.68 s) 
1427 (14.87 s) 
319 (3.39 s) 



Table 4.1: Performance evaluation for the images in Figure 4.1. The entries represent to the 
number of iterations and the CPU times in seconds, respectively, needed in order to attain a 
root mean squared error for the iterates below the tolerance e. 



Within this example we will focus on the anisotropic total variation function which 
is nothing else than the composition of the ^i-norm in y with the linear operator L. 
Due to the full splitting characteristics of the iterative methods presented in this paper, 
we need only to compute the proximal point of the conjugate of the Zi-norm, the latter 
being the indicator function of the dual unit ball. Thus, the calculation of the proximal 
point will result in the computation of a projection, which has an easy implementation. 
The more challenging isotropic total variation functional is employed in the forthcoming 
subsection in the context of an image deblurring problem. 

Thus, problem (4.1) reads equivalently 

inf \h(x) + q(Lx)\ , 

where h : W 1 — > R, h(x) = h\\x — b\\ 2 , is 1-strongly monotone and differentiable with 
1-Lipschitzian gradient and g : y — > R is defined as 3(2/1,2/2) = M\ {vi: Vz) 111- Then its 
conjugate g* : y — > K is nothing else than 



( Pi 


P2\ 


u 


x) 



9*(Pi,P2) = (A|| • ||i)* (pi,P2) = A 

where S = [—A, A] n x [—A, A]™. Taking xq S H, i>o £ y, 

VTT4^ ' 



70 G 0,min <^ 1, 



and ctq 



v l 2(l + 2p)^j; u 2 7o (l + 2 /3 )E™ill^l| 2 ' 

Algorithm 3.1 looks for this particular problem like 



(Vn > 0) 



Pi,n = x n -j n (x n -b + L*v n ) 

P2,n = V S {v n + Cr n Lx n ) 
V n +l = <7 n L(Pl,n ~ X n ) + P2,n 

X n +1 = JnL*(v n - p 2 ,n) + ln{x n ~ Pl,n) + Pi, 



1/^/1 + 2p7n(i - 7 n ), 7 n+ l = 9 n lr, 



>n+l 



Vnl®n- 



However, we solved the regularized image denoising problem with Algorithm 2.1, the 
primal-dual iterative scheme from [6] (see, also, [8]) and the accelerated version of the 
latter presented in [6, Theorem 2], as well, and refer the reader to Table 4.1 for a 
comparison of the obtained results: 
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(a) Noisy image, a = 0.06 




(b) Noisy image, a = 0.12 




(d) Denoised image, A = 0.07 




Figure 4.1: TV-l 2 image 
denoising. The noisy im- 
age in (a) was obtained af- 
ter adding white Gaussian 
noise with standard devia- 
tion a = 0.06 to the orig- 
inal 256 x 256 lichtenstein 
test image, while the out- 
put of Algorithm 3.1, for 
A = 0.035, after 100 itera- 
tions is shown in (c). Like- 
wise, the noise image when 
choosing a — 0.12 and the 
output of the same algo- 
rithm, for A = 0.07, after 
100 iterations are shown in 
(b) and (d), respectively. 



• ALG1: Algorithm 2.1 with 7 = small e > and by taking the last iterate 
instead of the averaged sequence. 

• ALG2: Algorithm 3.1 with p = 0.3, p = 1 and 70 = 2 n+tpW 

• PD1: Algorithm 1 in [6] with r = ^g, ra8 = 1 and by taking the last iterate 
instead of the averaged sequence. 

• PD2: Algorithm 2 in [6] with p = 0.3, r = 4p T a 8 = 1. 

From the point of view of the number of iterations, one can notice similarities be- 
tween both the primal-dual algorithms ALG1 and PD1 and the accelerated versions 
ALG2 and PD2. From this point of view they behave almost equal. When comparing 
the CPU times, it shows that the methods in this paper need almost twice amount of 
time. This is since ALG1 and ALG2 lead back to a forward-backward-forward split- 
ting, whereas PD1 and PD2 rely on a forward-backward splitting scheme, meaning that 
ALG1 and ALG2 process the double amount of forward steps than PD1 and PD2. In 
this example the evaluation of forward steps (i. e. which constitute in matix-vector 
multiplications involving the linear operators and their adjoints) is, compared with the 
calculation of projections when computing the resolvents, the most costly step. 

4.2 TV-based image deblurring 

The second numerical experiment that we consider concerns the solving of an extremely 
ill-conditioned linear inverse problem which arises in image deblurring and denoising. 
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For a given matrix A G M nxn describing a blur operator and a given vector b G M n 
representing the blurred and noisy image, the task is to estimate the unknown original 
image i£R™ fulfilling 

Ax = b. 

To this end we basically solve the following regularized convex nondifferentiable problem 
M n {\\Ax - bW, + XiTV^x) + A 2 Hxllj + ^(x)}, (4.2) 

where Ai, A2 G are regularization parameters and TV\ so : M n — > M. is the discrete 
isotropic total variation function. Notice that none of the functions occurring in (4.2) 
is differentiable, while the regularization is done by a combination of two regularization 
functionals with different properties. 

The blurring operator is constructed by making use of the Matlab routines imf ilter 
and f special as follows: 



H=fspecial ( ' gaussian ' ,9 ,4) ; % gaussian blur of size 9 times 9 

% and standard deviation 4 
B=imf ilter (X,H, 'conv ' , ' symmetric ' ) ; % B=ob s erv ed blurred image 

% X=original image 



The function f special returns a rotationally symmetric Gaussian lowpass filter of size 
9x9 with standard deviation 4, the entries of H being nonnegative and their sum 
adding up to 1. The function imf ilter convolves the filter H with the image X 
and furnishes the blurred image B. The boundary option "symmetric" corresponds to 
reflexive boundary conditions. Thanks to the rotationally symmetric filter H, the linear 
operator A defined via the routine imf ilter is symmetric, too. By making use of the 
real spectral decomposition of A, it shows that ||^4|| 2 = 1. 
For (y,z), (p,q) 6 3^, we introduce the inner product 

M N 
i=l j=l 



and define \\(y, z)\\ x = YaLi J2f=i yUij + z fj- One can check that || • || x is a norm on 
y and that for every x € W 1 it holds TV[ so {x) = ||Lx|| x , where L is the linear operator 
defined in the previous section. The conjugate function (|| • || x )* : y — > E of || • || x is for 
every (p, q) G y given by (see, for instance, [3]) 



(IHIx)*(p,ff) 

where 



0, if \\(p,q)\\x*<l 
+00, otherwise 



,q)\\x*= sup ((p,q),(y,z)} = max Jp\- + 

IK^)llx<l 1 1 g|^ V 



2 1 „2 

'J' 



Therefore, the optimization problem (4.2) can be written in the form of 

^f{f(x)+ gi (Ax)+g 2 (Lx)}, 
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(a) Original image 



(b) Blurred and noisy image (c) Reconstructed image 




Figure 4.2: TV-h-h image deblurring. Figure (a) shows the clean 256 x 256 cameraman 
test image, (b) shows the image obtained after multiplying it with a blur operator and adding 
white Gaussian noise and (c) shows the averaged sequence generated by Algorithm 2.1 after 400 
iterations. 



where / : R n ->■ M, f(x) = A 2 ||x||i + Vi]"( x )> 9i ■ ^ n ^ R, 9i(y) = h ~ Hi and 
g 2 : y -> K, ^(y,*) = Ai \\(v,z)\\ x . For every p G M n it holds gj(p) = <J[_i,i]»(p) + p T ft 
(see, for instance, [2]), while for any (p,q) £ y we have g^ip^q) = Ss{p,q), with 5 = 
{(Pj?) ^ 3^ : ||(Pj<z)||x* < We solved this problem by Algorithm 2.1 and to this 
end we made use of the following formulae for the proximal points involved in the 
formulation of this iterative scheme: 



Prox 7 /(x) = argmin (7A2H2H1 + = \\z — x\\ 2 \ = Vt w (x - 7A 2 l n ) Vx G 
ze[o,i] n L 2 * 

Prox 79 .(p) = argmin (7(2,6) + ~\\z - p\\ 2 \ = Pr-i.ii" (p - jb) \/p G R n , 
ze\-i,l] n I 2 J 



and 



Prox 79 *(p,g) = argmin - 1| (y, z) -(p,q)\\ 2 = V s {p,q) V(p,g) G ^, 



where 7 G l n is the vector in W 1 with all entries equal to 1 and the projection 

operator Vs '■ y — > S is defined as 



( /'-../ • '/' ./ 1 



max 



1,^^} max{l,^£} 
Taking x G H, (ui,o,U2,o) G M n x 3^, /3 = 7TT8 = 3, e G (0, ^) and ( 7 n)n>o 
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nondecreasing sequence in 



l-£ 



, Algorithm 2.1 looks for this particular problem like 



(Vn > 0) 



Pl,n = ^[0,1]™ ( X n ~ 7n (A*V 1>n + L*V 2) n + A 2 l n )) 
£>2,l,n = ^[-1,1]" ( v l,n + 7n(^n ~ &)) 
f>2,2,n = ^5 («2,ti + InLXn) 
V\, n +l = 7nA(j)i, n - X n ) + p 2 ,l,n 
^2,n+l = JnL(Pl,n ~ %n) + P2,2,n 

X n+ i = J n (A*(vi !n - P2,l,n) + £*(^2,n ~ P2,2,n)) + Pl,n- 

Figure 4.2 shows the original cameraman test image, which is part of the image process- 
ing toolbox in Matlab, the image obtained after multiplying it with the blur operator 
and adding after that normally distributed white Gaussian noise with standard devia- 
tion 1CP 3 and the image reconstructed by Algorithm 2.1 when taking as regularization 
parameters Ai = 3e-3 and A2 = 2e-5. 



4.3 TV-based image inpainting 

In the last section of the paper we show how image inpainting problems, which aim 
for recovering lost information, can be efficiently solved via the primal-dual methods 
investigated in this work. To this end, we consider the following TV-h model 

inf {\TVi so (x) + \\Kx -b\\i + S m n(x)}, (4.3) 

where A £ is the regularization parameter and TVj so : M n — >■ R is the isotropic 
total variation functional and K G M nxn is the diagonal matrix, where for % = 1, ...,n, 
Ki i = 0, if the pixel i in the noisy image b G M n is lost (in our case pure black) and 
Ka = 1, otherwise. The induced linear operator K : W 1 — > M n fulfills \\K\\ = 1, while, 
in the light of the considerations made in the previous two subsections, we have that 
TV iso {x) = \\Lx\\ x for all x G R n . 

Thus, problem (4.3) can be formulated as 

inf {f(x)+ gi (Lx)+g 2 (Kx)}, 

where / : W 1 -+ 1, f(x) = S [0Ar , gx : y R, <?i(yi,y 2 ) = || (yi,y%) ||x and g 2 : R n R, 
92(11) = \\y — b\\i- We solve it by Algorithm 2.1, the formulae for the proximal points 
involved in this iterative scheme been already given in Subsection 4.2. Figure 4.3 shows 
the original fruit image, the image obtained from it after setting to pure black 80% 
randomly chosen pixels and the image reconstructed by Algorithm 2.1 when taking as 
regularization parameter A = 0.05. 
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(c) Reconstructed image 
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